Method for dynamically training a system to determine an age rating of media content

ABSTRACT

A system and method for dynamically training a system to determine an age rating for media content. An exemplary method includes obtaining age rating data for a plurality of territories; determining, based on the age rating data, a similarity vector relating to the target territory; determining, for the similarity vector, a territory associated with a highest prediction score; in response to determining that the territory associated with the highest prediction score is not the source territory, generating a training dataset comprising the age rating data for the target territory, the source territory, and the territory associated with the highest prediction score; and executing a machine learning model, trained by the training dataset, to output an age rating for a content item in the target territory based on an age rating for the content item in the source territory.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to GB Application No. 2115955.3, filedon Nov. 5, 2021, the contents of which are herein incorporated byreference.

FIELD OF TECHNOLOGY

The present disclosure relates to a method of converting an age ratingof media content from one country to another.

BACKGROUND

Media content is often classified with an “age rating” that providesguidance on the appropriate audience for that media content. Forexample, a particular film may be classified as a “15” in the U.K.,meaning that it is suitable for audience members aged 15 and above. Theprocess of deciding on a particular age rating in the U.K. takes intoaccount the specific content of the media, such as the degree ofviolence, use of language, etc., as well as any other culturalsensitivities in the U.K. Sensitivities may be particular words, from apre-determined list of words, displayed alongside an age rating asadditional information relevant to the age rating. For example, in theU.K. words such as “strong violence”, “threat” and “bad language” may bedisplayed as sensitivities alongside a particular age rating. Instead ofwords or symbols, other formats of displaying sensitivities may be usedby different countries. The sensitivities may be explicitly defined andregulated in certain countries, whereas in others, they may not.

While a system for classifying media content in one country can be basedon a set of rules, different countries or different cultures may have adifferent viewpoint. For example, something that is deemed to beoffensive in one country may not be deemed to be offensive in anothercountry. As a result, a particular age rating for one country may not bedirectly mapped to a suitable age rating for another country.Furthermore, even if the viewpoint on the age rating is shared betweencountries, different countries use different classification systems inthat a U.K “15,” for example, may not have an equivalent age rating inanother country.

Currently, the process of deciding an age rating for a specific countryis predominantly a manual process, whereby media is individuallyclassified for each country based on the specific classification systemand sensitivities for that country. Not only is this slow and labourintensive, but it can also result in inconsistency and human error. Alack of publicly available data further exacerbates this problem byrendering traditional statistical methods inaccurate since often theyrely on learning trends from large amounts of data. There is a need fora quick and efficient process for converting an age rating and/or listof sensitivities from one country to another country.

Because there is no universal committee that determines ratings for allmedia content (e.g., some local movies in a particular territory may notbe given a rating from a committee in a different territory), parentalcontrols are not as effective as they can be on a global scale. Parentalcontrol options are commonly part of security/privacy settings indigital platforms such as streaming services. Due to global mediaconsumption, parental control settings may be unable to effectivelyblock restricted content for an audience. For example, a movie on astreaming service such as Netflix that would be blocked in America maybe available to the same user if the user travels to Europe (even thoughit should remain blocked). On the other hand, a movie that should not beblocked (e.g., a PG-13 movie) may be blocked inadvertently. This causesaccess issues and compromises the effectiveness of parental controls.

SUMMARY

Provided herein is a computer-implemented method of training a systemfor predicting, based on an age rating of a content item in a sourceterritory, an age rating of a content item in a target territory. Themethod comprises: obtaining age rating data for a plurality ofterritories, the age rating data comprising a plurality of age ratingsfor a respective plurality of content items for the source territory,the target territory and at least one other territory; determining,based on the age rating data, a similarity vector relating to the targetterritory, the similarity vector comprising a prediction score for eachof the source territory and the at least one other territory;determining, for the similarity vector, one or more highest predictionscores; providing, as input data to a machine learning model, the agerating data for the target territory, the territory associated with thehighest prediction score, and, when the territory associated with thehighest prediction score is not the source territory, the sourceterritory; and training the machine learning model using the input datato provide the age rating for the content item in the target territory.

Optionally, the content item comprises media content.

Optionally, the respective plurality of content items comprisesthousands of content items.

Optionally, the at least one other territory comprises a plurality ofterritories.

Optionally, the step of determining a similarity vector comprisesgenerating a similarity matrix comprising a plurality of similarityvectors for a plurality of pairs of territories.

Optionally, the method further comprises clustering the plurality ofsimilarity vectors into clusters.

Optionally, the step of determining one or more highest predictionscores comprises: (a) selecting one or more prediction scores exceedinga threshold, or (b) selecting a predetermined number of predictionscores having a highest value, or (c) selecting prediction scores withina set number of standard deviations above a mean prediction score.

Also provided is a computer-implemented method for predicting, based onan age rating of a content item in a source territory, an age rating ofa content item in a target territory using the trained system. Themethod comprises providing, as an input to the system, an age rating fora content item in a source territory; and outputting, as an output ofthe system, an age rating for the content item in the target territory.

Also provided is a computer system comprising one or more processors andone or more non-transitory computer-readable media storing instructionsthat, when executed by the one or more processors, cause the one or moreprocessors to perform the method.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative implementations of the present disclosure will now bedescribed, by way of example only, with reference to the drawings. Inthe drawings:

FIG. 1 schematically shows a process of generating training data fromraw data for use to train a model to determine age ratings for a targetcountry.

FIG. 2 schematically shows a process of generating prediction scores forthe similarity clustering.

FIG. 3 illustrates the relationship between the ratings in the U.K. andFrance.

FIG. 4A illustrates an example similarity matrix, 4B shows a territorysimilarity matrix, 4C shows a territory—age rating similarity matrix, 4Dshows a territory—age rating+sensitivity similarity matrix.

FIG. 5 schematically shows a process of clustering.

FIG. 6A shows a high-level architecture of a multitask learning model,6B shows a model trained for territory 1, 6C shows an examplearchitecture of multitask learning model 6A.

FIG. 7 schematically shows a process of determining age ratings for atarget country.

FIG. 8 illustrates a block diagram of one implementation of a computingdevice.

DETAILED DESCRIPTION

Exemplary In overview, described herein are methods for determining anappropriate age rating for a content item. In this context, contentitems are any type of media content such as video, audio or picturecontent. Specific examples are films, documentaries, television programsand the like. These methods are used in the scenario in which it isdesired to predict an appropriate age rating for a particular contentitem in a particular territory, according to the age ratings system ofthat particular territory. These methods have additional use inpredicting appropriate territory specific sensitivities for a particularcontent item in a particular territory according to the age ratingssystem of that particular territory.

To determine an appropriate age rating for a particular content item ina particular territory, content data associated with both the particularcontent item and with a source territory is input into a trained machinelearning model. The trained machine learning model has been trained toreceive content data associated with both a content item and with a“source territory”, and output an age rating for the particular contentitem in the particular territory. The source territory being a territoryfor which at least some content data for that particular content item isalready known. The content data for the source territory may be, forexample, an age rating of the content item. This may be termed a sourceterritory age rating. For example, it may be known that a particularfilm has a U.K. age rating of 12. In this example, the content item is afilm, the source territory is the U.K. and the source territory agerating is 12. Continuing the example, it may be desired to know what theage rating would be for this content item in a target territory, such asFrance. In other words, it may be desired to know the target territoryage rating.

The age rating system of the target territory may not exactly map to theage rating system of the source territory. For example, while the U.K.may be an age rating of “12”, in France a “12” may traditionallycorrespond to a different, higher age rating in the U.K., such as a“15”. Even if the age rating systems of the source and targetterritories are the same, local sensitivities may still mean that whatis regarded as a certain age rating on one territory would not beregarded as the same age rating in another territory. Localsensitivities are themes, images, depictions and other criteria which,due to local reasons such as religion or culture, are deemed to be moreor less suited to a higher or lower age bracket. For this reason, theprocess may also include additional content data for the sourceterritory as the starting point, such as text data associated with thecontent item. Such text data may include at least one of a textdescription of the content item and a list of sensitivities related tothe content item. The list of sensitivities may comprise knownsensitivities associated with the source territory, and may optionallyor instead comprise one or more custom sensitivities or “tags” for thesource territory. Tags are sensitivities associated with the contentitem that are not necessarily specific to the source territory, but mayhowever be impactful in another territory. For example, certainterritories may regard “national conflict” as an impactful sensitivitywhereas the U.K. does not. “National conflict” would not traditionallybe a sensitivity associated with a content item if the source territorywas the U.K. (due to having limited or no impact on a U.K. age rating),however, if the content item contained “national conflict”, the sourceterritory may still carry the tag “national conflict” as an additionalsensitivity. Such tags may therefore be part of age rating data toimprove the training of a machine learning model, since sensitivitiesthat only affect some territories may nevertheless be included in theage rating data for other territories.

The processes described herein enable an accurate and automaticdetermination of an appropriate target territory age rating for aspecific content item based on known content data for a sourceterritory. To achieve the accurate and automatic determination of anappropriate target territory age rating, one or more machine learningmodels are trained to predict the target territory age rating.

Before training the one or more machine learning models, the raw data ispre-processed. The raw data may be obtained from any appropriate sourcesuch as publicly available databases or data supplied from thirdparties. The raw data comprises raw data from a plurality ofterritories. For example, the raw data may comprise raw data fromterritory A and raw data from territory B. Territory A may be used asthe source territory in inference, and territory B may be the targetterritory in inference. In this case therefore, the raw data isprocessed to create training data for training a machine learning modelto output a target territory age rating for a target territory B basedon input content data of a source territory A. In an example, the rawdata includes a list of all territory A age ratings for which territoryB age ratings are known for a particular content item. Such territory Aand B age ratings in this raw data are known due to previous methods ofdetermining such age ratings, such as manual methods. Optionally, theraw data further includes any territory A sensitivities and anyterritory B sensitivities for all the content items in the dataset. Itshould be noted that in some cases, territories do not carry asensitivity associated with an age rating. Variants or combinations ofthese examples are of course possible and it is understood that theseexamples are used for illustrative purposes.

The raw data is pre-processed to present the raw data in vector form.Pre-processing comprises encoding the raw data. Age ratings of eachcontent item may be encoded using one-hot encoding, the text descriptionof each content item may be encoded using Term Frequency-InverseDocument Frequency (TF-IDF) encoding, and the list of sensitivities ofeach content item may be encoded with categorical encoding. Alternativeknown encoding methods could also be used in order to represent the rawdata in vector form, as would be understood by the skilled person, andthe exact encoding method is not essential.

The resulting encoded vectors are concatenated into a single encodingvector for each territory associated with each content item.Pre-processing may further comprise additional known natural languageprocessing techniques such as stop word removal, stemming and datacleaning to remove duplications and abnormalities. Furthermore, thepre-processed data may be up-sampled to increase the amount of data. Inthis case, the amount of “recent” data is increased. This involves theduplication of content items such that there is a higher probabilitythat more “recent” content items are duplicated. For example, moreweight may be placed on content items created within a certain timethreshold, for example within the last two years. Beneficially, theup-sampling of more recent data compensates for the problem arising dueto classification criteria changing over time and ensures the model canaccurately determine age ratings according to more recent classificationcriteria.

In one implementation, a machine learning model is trained using thepre-processed data from only two territories: A and B. When the amountof training data for territories A and B is high (e.g., above athreshold size), this model can accurately predict the age rating interritory B for a particular content item. Such a prediction is based onan input comprising content data associated with both the particularcontent item and with territory A. A high amount of training data may beupwards of 5000 content items, for example. In another implementation, amachine learning model is trained to accurately predict sensitivitiesfor the particular content item in territory B. Such a prediction isalso based on an input comprising content data associated with both theparticular content item and with territory A. Whilst a predictionaccuracy of over 80% is achieved with large amounts of data, both modelimplementations become less effective as the amount of data decreases.Furthermore, since the requirement for such a model arises partiallyfrom the lack of age-rating data for content items in other territories,typically, there is insufficient data to train the implementationsdescribed above to produce accurate age rating or sensitivity data fortarget territories.

Further modifications to the pre-processed data solve the problem ofinsufficient publicly available data, allowing accurate predictions tobe achieved even with small amounts of training data. In addition, evenwhen high amounts of data are available, these modifications furtherincrease the accuracy of the machine learning model, allowing ageratings to be predicted with over 90% accuracy for target territories.

FIG. 1 schematically depicts a process 100 of generating training datafrom data pre-processed as described above. Such training data is usedto train a machine learning model to output a target territory agerating for a content item, based on input content data of a sourceterritory for the content item. Process 100 can be implemented by anykind of general computer or computing device and is specifically suitedfor computers with a processing capacity adapted for machine learningprocesses.

In step 102, similarity vectors are determined. Similarity vectors aremade up of two or more prediction scores. Based on the pre-processeddata, a prediction score may be calculated to quantify how well an agerating in territory B can be predicted from the appropriate age ratingin territory A. For example, for a particular film content item, the rawdata includes that the age rating in the U.K. (territory A) is “12”,whereas the age rating in France (territory B) is “TP”. The predictionscores may further quantify whether these predictions can be improvedwhen sensitivities are taken into account. The pre-processed dataincludes a number of examples such that a dataset of particular contentitems and their respective age ratings in both territories is provided.

A process 200 for determining the similarity vector in step 102 of FIG.1 will now be described with reference to FIG. 2 . It is noted that,although FIG. 2 describes the process of generating territory similarityvectors, the process described in FIG. 2 may also be used to generateadditional types of similarity vectors as described in more detailbelow. The prediction scores are generated pairwise such that for eachpair of territories A and B, a score is generated. In order to generatea prediction score for territories A and B, all content items (e.g.specific films) that have previously-determined age ratings for bothterritories A and B in the pre-processed data are identified in step202.

In step 204, for each different age rating from territory A, a mostfrequently occurring age rating in territory B is determined. Thepre-processed data may not be a 1:1 mapping of age ratings betweendifferent territories. This may be for various reasons. For example,territory A may have an age rating system comprising a first number ofpossible different age ratings “n” for content items (e.g. 8), whereasterritory B may an age rating system comprising a second, differentnumber of possible different age ratings for content items (e.g. 6).Therefore, a most frequently occurring (“MFO”) age rating for territoryB determined for each territory A age rating. This will be explainedfurther in relation to FIG. 3 .

FIG. 3 shows an example mapping of age ratings between two differentterritories. As is clear in the context of the described examples,“mapping” may refer to the correlation of the content, based on metadataand source territory age rating, to a destination territory's agerating. In this example, the left side of the graph represents territoryA (U.K.) and the right side of the graph represents territory B(France). As can be seen, most, but not all, of the content items havingan age rating of “12” in the U.K. have an age rating of “TP” in France.In the specific example of FIG. 3 , 656 out of 666 U.K. “12” age ratingsin the pre-processed data correspond to an age rating of “TP” in France,whereas 10 out of 666 U.K. “12” age ratings correspond to other ageratings in France, such as “12” or “16”. Thus, “TP” is the mostfrequently occurring age rating in France for a “12” age rating in theU.K., and therefore “TP” is regarded as the most frequently occurringage rating in France for the age rating “12” in the U.K.

Similarly, it can be seen that most of the content items having an agerating of “15” in the U.K. also have an age rating of “TP” in France. Inthe specific example of FIG. 3 , 671 out of 971 U.K. “15” age ratings inthe pre-processed data correspond to an age rating of “TP” in FranceThus, “TP” is also regarded as the most frequently occurring age ratingin France for the age rating “15” in the U.K.

Returning to FIG. 2 , in step 206, for each of the n different ageratings for territory A, the number of content items corresponding totheir most frequently occurring age rating in territory B is summed anddivided by the total number of content items—this is the predictionscore_(AB) as shown below in equation 1.

$\begin{matrix}{{{Prediction}{score}_{AB}} = \frac{\begin{matrix}\left( {{A\_ AgeRating1}_{MFO} +} \right. \\\left. {{{A\_ AgeRating2}_{MFO}\ldots} + {\cdots{A\_ AgeRating}n_{MFO}}} \right)\end{matrix}}{\begin{matrix}\left( {{A\_ AgeRating1}_{total} +} \right. \\\left. {{{A\_ AgeRating2}_{total}\ldots} + {\cdots{A\_ AgeRating}n_{total}}} \right)\end{matrix}}} & (1)\end{matrix}$

Where A_AgeRating 1_(MFO) to A_AgeRating n_(MFO) are, for each of the1-n different age ratings in the age rating system of territory A, thenumber of content items that correspond to their most frequentlyoccurring age rating in territory B, and where A_AgeRating 1_(total) toA_AgeRating n_(total) are the total number of content items having eachage rating of territory A. Using the example discussed above for theU.K. and France, the prediction score is calculated by summing, for eachof the different U.K. age ratings, the number of U.K. age ratings thatcorresponds to their respective most frequently occurring France agerating, and dividing that sum by the total number of content items inthe pre-processed data that have both France and U.K. age ratings.

In this example, this is (656+671+ . . . )/(666+971+ . . . ). Althoughthe above example has only described the scenario of U.K. “12” and U.K.“15” age ratings, the process would be completed for all U.K. ageratings, as represented by the “ . . . ” in the formula. The outcome isa decimal number between 0 and 1 representing a percentage of how well aU.K. age rating predicts an age rating in France, where 1 indicates 100%correspondence of all age ratings.

In step 208, a territory similarity matrix is generated from theprediction score for each pair of territories.

FIG. 4A shows an example of a generic matrix, H with m rows and ncolumns. Each element h_(ij) describes the element in the ith rows andthe jth column where i takes any value from 0 to m and j takes a valuefrom 0 to n.

FIG. 4B shows an example of a territory similarity matrix having thestructure of FIG. 4A, where each element h_(ij) describes how well arating in territory T_(j) is predicted by the rating in territory T_(i).Territories A and B, previously described, would be examples of one ofthe territories T_(i) and T_(j) respectively. The first column can beinterpreted as follows: All age ratings of a content item in Argentina(1.00) correspond to their most frequently occurring age rating inArgentina (exact correspondence), 68% (0.68) of all age ratings inAustralia correspond to their most frequently occurring age rating inArgentina, 50% (0.50) of all age ratings in Austria correspond to theirmost frequently occurring age rating in Argentina, 48% (0.48) of all ageratings in Brazil correspond to their most frequently occurring agerating in Argentina. Each element of each column of the territorysimilarity matrix therefore describes how well a particular territoryT_(i) predicts the age ratings of the particular territory T_(j)corresponding to that column. Thus, each column is referred to fromhereon as the similarity vector for that column's territory. Forexample, the first column of FIG. 4B shows the similarity vector ofArgentina being [100, 68, 50, 48] describing how well other countriespredict the age rating in Argentina. In step 210, similarity vectors areobtained from each column of the similarity matrix.

Returning to FIG. 1 , after the matrix of prediction scores has beengenerated in step 102, as described above, in step 104, the highestprediction scores in each column are identified. These scores indicatethe territories that best predict the territory T_(j) of that column.The best predicting territories may be selected by an appropriateindicator, for example, by using a threshold value for the predictionscore, such as 90%. Alternatively, the top 3 territories may beselected, or the territories with prediction scores of a set number ofstandard deviations above the mean prediction scores for that column.These territories are then used to train a common part of the machinelearning model for that territory T_(j), as described below in moredetail with reference to FIG. 6 . If the predictive accuracy of themodel trained using this data is high, for example over a particularthreshold, steps 106-110 may be discarded.

Alternatively, a higher accuracy may be obtained using steps 106-110. Instep 106, the similarity vectors obtained from each column of theterritory similarity matrix are clustered into clusters. These clustersgroup territories that are predicted by similar combinations ofterritories. In the described embodiment, trial-and-error is used todetermine which clustering techniques provide the best results. Forexample, one clustering technique may be used and, depending on theaccuracy of the outcome from the trained model, another clusteringtechnique may be used. This process may be repeated to find the mostaccurate clustering method. For example, top K and K-means clusteringmay be used.

FIG. 5 schematically shows a general process of clustering. It isunderstood that any appropriate clustering technique could be used inaddition to or instead of the techniques described above. The clustersobtained in step 106 comprise the combination of territories used totrain the machine learning model to determine age ratings for a targetterritory. The combination of territories may be any combination ofterritories, and not merely the examples described above. The term“clusters” as used in this specification refers to two or more clusters(groups) generated from a process of clustering three or more vectors.In step 502, the cluster centres are initialised. In step 504,assignments to the clusters are computed. In step 506, the loss isdetermined. The clustering process 500 iterates between steps 504 and506 in order to minimise the loss until some pre-determined threshold isreached.

Each of the resulting clusters is a group of territories predicted bysimilar combinations of territories. The territories with the highestprediction scores for territory T_(j), obtained from step 104, are addedto the cluster in which the vector obtained from territory T_(j) isgrouped. The resulting cluster is a cluster of territories that eitherpredict territory T_(j) or are predicted by similar territories toterritory T_(j) (are similar to territory T_(j)).

In step 108, cluster-specific data preparation is performed. For eachcluster from step 106, the territory with the largest number of contentitems is identified and the content items in the other territories inthe cluster are up-sampled, so that each territory has the same numberof content items as the territory with the largest number of contentitems. For example, say the similarity cluster for Argentina includes:France with 10 content items, Brazil with 5 content items and Australiawith 7 content items. In this case, the content items in Brazil andAustralia are up-sampled so each territory in the similarity cluster forArgentina has 10 content items.

As described above in the pre-processing section, up-sampling comprisesthe duplication of territory content items. Following the above, 3content items from Australia may be randomly selected and these contentitems duplicated, and 5 content items from Brazil may also be randomlyselected and duplicated. The result is that each territory of thesimilarity cluster now contains 10 content items. Alternative methods ofup-sampling, such as generating content items that represent acombination of the data from other content items from a certainterritory could also be used.

In step 110, the resulting cluster may be used to train the machinelearning model. Should it be desirable to further increase modelaccuracy, the cluster may be added to by performing process 200 in orderto obtain additional similarity matrices and therefore additionalsimilarity vectors. After every addition to the cluster, the model isretrained and the accuracy of the model obtained in order to find thecombination of territories that produces the most accurate model foreach territory.

Obtaining additional similarity matrices is particularly useful when theterritory prediction score between two territories is low or when thetrained model gives a low accuracy of result for example, under 80%accuracy of model). For each additional matrix, prediction scores aregenerated as per the process described above with reference to steps202-210 but with the differences described below.

A first additional similarity matrix, an territory-age rating similaritymatrix, provides insight into whether a particular territory T_(i) agerating corresponds to a specific territory T_(j) age rating despite thetotal percentage of age ratings corresponding to their most frequentlyoccurring age rating in territories T_(i) and T_(j) being low. Forexample, only 40% of the content items in Columbia have age ratingscorresponding to their most frequently occurring age rating in Germany,but there is a specific age rating in Columbia where 80% of the examplesof that age rating correspond to a German:16. As described above withreference to step 106, any additional similarity matrices may be used inclustering. Furthermore this information may be used later on whenincluding optional adding hard coded rules to the trained machinelearning model.

Each element h_(ij) describes how well a particular rating (specified inthe matrix) in territory T_(i) predicts a rating in territory T_(j).Territories A and B, previously described, would be examples of one ofthe territories T_(i) and T_(j) respectively. FIG. 4C shows an exampleof a territory-age rating similarity matrix. The first column of FIG. 4Cindicates that 69% of the examples of age rating MA15+ in Australia havea corresponding age rating in Italy, 100% of the examples of age rating12A in Ireland have a corresponding age rating in Italy, 60% of theexamples of age rating 16 in Germany have a corresponding age rating inItaly and so on. This shows that there is an age rating in Italy that isperfectly predicted by both Ireland: 12A and a rating in Italy that isperfectly predicted by Hong Kong: 11A. Therefore, it is likely that theaddition of these territories to the similarity cluster used to trainthe model for Italy may improve the accuracy of the model.

A second additional similarity matrix indicates the prediction scorewhen particular sensitivities are associated with a specific age rating.This provides a further level of detail and indicates the similaritybetween combinations of age ratings and sensitivities FIG. 4D shows anexample of a territory-age rating+sensitivity similarity matrix. Thefirst column of FIG. 4D indicates that 66% of the examples of age ratingMA15+ in Australia that have drug use have a corresponding age rating inItaly, 82% of the examples of age rating MA15+ in Australia that havesex references have a corresponding age rating in Italy, 60% of theexamples of age rating 16 in Germany that have strong language have acorresponding age rating in Italy and so on. Again, particularly highprediction scores here can also be used to choose specific territoriesto add to the similarity cluster used to train the model for aparticular territory.

FIG. 6A shows a high-level architecture of a multitask learning model600. As this skilled person would understand, multitask model is a termused in the art to describe a machine learning model in which multiplelearning tasks are solved at the same time, while exploiting both thecommonalities and differences across tasks. The multitask learning modelmay comprise two parts: an optional first part comprising common layers(610) and a second part comprising specific layers (620, 630, 640). Asdescribed above, the implementation of the model which does not requirethe steps 102-110 described in FIG. 1 due to a high or sufficientinitial model accuracy, which is specifically trained with twoterritories only, does not require a common part and is trained purelywith the territory specific data.

The first part is made up of layers of the network which are sharedbetween all the territories in a cluster and trained to minimise theloss over all the shared territories. A source territory is provided inaddition to the territories in the cluster for training. Here, thecluster is obtained in the process described above with reference toFIGS. 4A-5 . The specific part is made up of layers of the network whichare trained only to predict the most accurate rating for the specificterritory T_(j). It is this specific part which outputs the predictionfor the model. There may be any number of specific layers forterritories 1 to N. For example, a cluster of N territories is obtainedusing the process described above. Element 620 comprises territory 1specific layers and makes up the second part of a multiclass model forterritory 1, element 630 comprises territory 2 specific layers and makesup the second part of a multiclass model for territory 2, and element640 comprises territory N specific layers and makes up the second partof a multiclass model for territory N. The resulting model for eachterritory comprises (1) common layers trained using the territories inthe territory specific cluster, and (2) territory specific layers forthat territory.

FIG. 6B shows an example model trained to provide a prediction forterritory 1. FIG. 6A describes an implementation where at least some of,and possibly all of, the steps described in FIG. 1 were performed toobtain additional territories for training the model. It is understoodthat in an implementation where these steps are not performed, themodel. only comprises specific layers for a particular target territoryand is trained on data relating to the source and target territory suchthat the model is trained to predict an age rating or list ofsensitivities for the target territory given only information about thesource territory as an input.

FIG. 6C shows an example architecture of the multitask learning model600 described above with reference to FIG. 6A. In the specificimplementation summarised by FIG. 6C, each part of the multitasklearning model 600 comprises one or more layers.

The first part comprises an input layer 611 followed by alternatingdense (first and second dense layers) layers 612, 614 with ReLuactivation function and MaxNorm kernel constraint of 3, and dropout(first and second dropout layers) layers 613, 615 with dropout rate of0.2 (20%), on a scale from 0 to 1 where 0 indicates no dropout and 1indicates no output from the layer.

The terms dense layer may also be referred to in the art as fullyconnected layer and is only briefly described here for the sake ofbrevity. Dense layers or fully connected layers refer to layers whereall the neurons in the previous layer are connected to all neurons inthe next layer. Dropout layers are layers that ignore a number ofneurons in the output which has the effect of making the layer act likea layer with a different number of neurons to the prior layer. An effectof using dropout layers is that neurons in a layer are prevented fromsynchronously optimizing their weights, which is known to overcome thecommon problem of overfitting in machine learning.

The second part comprises a third dropout layer with dropout rate of 0.5(50%) in between a third and fourth dense layer, the third dense layerwith ReLu activation and MaxNorm kernel constraint of 5 and the fourthdense layer with Softmax activation function to generate the finalprediction. The terms activation, ReLu, Softmax, kernel and MaxNorm areknown in the art and are as would be understood by the skilled person.

Although the described implementation describes the same architecture ineach of the specific parts, it should be understood that each specificpart may have entirely different architecture and the architecture thatprovides the highest accuracy for each specific territory is chosen. Forexample, some territories may produce higher accuracy with very complexmodels and some with simple models.

The multitask learning model 600 may be trained to determine age ratingsfor a target country or sensitivities for a target country.

The age rating prediction model is trained to predict the age ratingthat a content item will have in the target territory for a given theinput vector for a specific content item obtained from a sourceterritory. The model outputs a probability of the content item being ineach age rating for the target country. The highest probability is usedfor the actual prediction. For this model, all the similarity vectorsobtained from the matrices in FIGS. 4B-4D may be used in obtaining theterritories for training the common layers of the model.

The sensitivity prediction model is trained to predict the list of localsensitivities for the target territory given the input vector for aspecific content item. This model performs a multi-label classificationand can predict multiple sensitivities that apply for the example.

The age rating prediction model is trained to minimise the CategoricalCross-entropy loss function. The Categorical Cross-entropy loss functionminimises the difference between the target value y_(i) and the modeloutput,

as described in equation 2:

$\begin{matrix}{{Loss} = {\overset{n}{\sum\limits_{i = 1}}{y_{i}{\log\left( \hat{y_{i}} \right)}}}} & (2)\end{matrix}$where n is the total number of categories, for example age ratings, inthe model output. An Adam optimiser is used for gradient descent.

The sensitivity prediction model is trained to minimise the BinaryCross-entropy loss function. A specific implementation may therefore usea Sigmoid activation function in the last layer of the model, instead ofthe Softmax activation described above with reference to the age ratingprediction model. However, it is understood that any activation functioncompatible with a binary cross entropy may also be used.

The Binary Cross-entropy loss function minimises the difference betweenthe target value y_(i) and the model output,

, where each sensitivity, m can have one of two values, 0 or 1,indicating whether or not they are applicable to a particular contentitem, as described in equation 3:

$\begin{matrix}{{Loss} = {{{- \frac{1}{m}}{\overset{m}{\sum\limits_{i = 1}}{y_{i}{\log\left( \hat{y_{i}} \right)}}}} + {\left( {1 - y_{i}} \right){\log\left( {1 - \hat{y_{i}}} \right)}}}} & (3)\end{matrix}$

The multitask model of FIG. 6A is trained for a specific territory bytraining both the common and specific parts of the model iteratively foreach of the territories in the similarity cluster for that specificterritory. For example, in a single epoch the model trains the commonpart and the specific part for territory 1 by going once over all thecontent items in territory 1, in the next epoch the model trains thecommon part and the specific part for territory 2 by going once over allthe content items in territory 2, in the n^(th) epoch, the model trainsthe common and specific part for territory n by going once over all thecontent items in territory n and the process is repeated until the aaccuracy threshold is met or the optimal accuracy is reached.

This results in a model that learns from territories that are similar tothe target territory, not just the target territory since the commonpart of the model is updated based on its ability to predict the agerating for each of the territories in the cluster, not just the targetterritory. Beneficially, this approach effectively increases the amountof data available to the model for training. As such, this approachsolves the problem of providing high prediction accuracy when the amountof raw data for the target territory itself is otherwise insufficient.After training, the specific parts generated for the other territoriesin the cluster are discarded, leaving only the common part and thespecific part for each territory for use in inference.

Optionally, manual rules may be applied and used in conjunction with thetrained multitask learning model. The manual rules reflect knowninformation on the classification methods in different countries, whichmay modify or override the prediction of the trained machine learningmodel.

Age ratings and sensitivities of the target territory can be obtainedfrom the appropriately trained model by the method described in FIG. 7 .Taking the age rating prediction model as an example, in step 702 a, thecontent item for which an age rating is to be determined is provided asinput to the age rating prediction model trained for the targetterritory.

In step 702 b, the data describing that content item, relating to asource territory, is input into the model. This data may be manuallyprovided by the user as raw data, obtained from a third party as raw orencoded data or obtained from a database in the system (described belowwith reference to FIG. 9 ) in the encoded vector form described above inthe pre-processing step. The data describing the content item is encodedinto a vector, if not yet in vector form, and in step 704 is provided tothe trained model for inference. In step 706, the model outputs an agerating classification for the content item in the target territory.

The above-described methods provide various advantages. Such methodsprovide an accurate and efficient manner of predicting an appropriateage rating for a particular content item in a particular territory. Asthe method relies on a trained machine learning model that has beentrained using age rating data for a plurality of territories, the methodtakes into account not only data from a source territory and a targetterritory, but also at least one other territory which may provide moreaccurate age rating predictions for the target territory. Includingadditional territories in this manner greatly improves the accuracy ofthe method and provides a prediction accuracy that is otherwise notpossible using manual techniques. Beneficially, the automation of theprediction removes human error in determining an age rating for thetarget territory, as only the age rating for the content item in thesource territory is needed as an input to the trained model.

Further, the method is effective in the scenario where age rating datafor a particular target or source territory is limited. This is becausethe method makes use of data not only relating to the target and sourceterritory, but also relating to at least one other territory. As such,the age rating data from multiple territories comes together as part ofthe model to, in aggregate, provide a large data size for predicting anappropriate age rating in the target territory. Another benefit of thedescribed methods is that appropriate sensitivities, and not just ageratings, may also be predicted automatically.

Because the system generates training datasets based on the predictionscores, the machine learning model trained on the custom-built trainingdatasets is more accurate. For example, in response to determining thatthe territory associated with the highest prediction score is not thesource country, the system may generate a training dataset comprisingthe age rating data for: the target territory, the source territory, andthe territory associated with the highest prediction score. Here, theterritory associated with the highest prediction score may serve as anintermediary territory. For example, there may not be enough data toaccurately map age ratings between the source territory and the targetterritory. However, the intermediary territory may have a more accuratemapping of age ratings with the source territory and the targetterritory (evidenced by the prediction score). A training dataset thatincludes all three territorial age rating data is generated to allow, ona high level, a mapping from the source territory to the intermediaryterritory and then a mapping from the intermediary territory to thetarget territory. On the other hand, the system may determine that theterritory associated with the highest prediction score is the sourcecountry, and may generate the training dataset using the age rating datafor only the target territory and the source territory. The benefits ofa more focused training dataset include faster training times, fewerfalse positives, and less use of storage.

Subsequent to successfully training the machine learning model, themodel can be used for a variety of tasks. For example, ratingscommittees from different territories can execute the machine learningmodel to received age rating conversion recommendations. Because thetraining dataset of the machine learning model accounts for age ratingdata from a plurality of territories, each with their own ratingscommittee, the machine learning model's recommended output can push ageratings to be more consistent across different territories.

In another aspect, the machine learning model may be used to implementparental controls in different territories. For example, media contentplatforms such as streaming services and cable often provide parentalcontrol options to restrict content from being viewed (e.g., preventchildren from viewing violent movies). Suppose that a user accesses astreaming service that is available in different parts of the world orprovides access to movies/shows from different parts of the world (i.e.,the media content platform is accessible in a source territory and atarget territory). A user may be interested in viewing a local moviethat has a “18” rating in the U.K. and has not been rated in France. Theuser may be accessing a version of the streaming service associated withFrance (e.g., while in France or while using a virtual private network(VPN) in a different country to access content in France). The parentalcontrol threshold may be set in a ratings scale associated with France.For example, the parent control threshold may be set to “TP” such thatall other movies/shows are blocked. Here, the U.K. rating “18” cannot bedirectly compared to the France-based parental control threshold andneeds to be converted.

A system (e.g., a parental control module of the media content platform)can be provided that is configured to determine, by executing thetrained machine learning model, that the input content item (e.g., thelocal movie being requested for access) has a first age rating in thetarget territory (e.g., “12” rating in France), wherein the first agerating (e.g., “12”) corresponds to a second age rating of the inputcontent item in the source territory (e.g., “18” in the U.K.). Inresponse to determining that the first age rating (e.g., “12”) exceeds aparental control threshold (e.g., “TP”), the system may generate analert indicative of a parental control when the input content item isrequested for access. In some aspects, the alert is generated on agraphical user interface of the media content platform. For example, thealert may warn the user that he/she is attempting to access restrictedcontent. In some aspects, subsequent to generating the alert, the systemmay block the request to access the input content item to fully executea parental control.

Because the parental control threshold and the first age rating are setin a first rating scale of the target territory, and the second agerating is set in a second rating scale of the source territory that isincompatible for direct comparison with the parental control threshold,the execution of the machine learning model is necessary. Otherwise,movies that cannot be compared with the parental control threshold maybe accessible when they should not be, or movies that should beaccessible may be prematurely blocked to avoid parental controlthreshold issues.

FIG. 8 illustrates a block diagram of one implementation of a computingdevice 800 within which a set of instructions, for causing the computingdevice to perform any one or more of the methodologies discussed herein,may be executed. In alternative implementations, the computing devicemay be connected (e.g., networked) to other machines in a Local AreaNetwork (LAN), an intranet, an extranet, or the Internet. The computingdevice may operate in the capacity of a server or a client machine in aclient-server network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The computing devicemay be a personal computer (PC), a tablet computer, a set-top box (STB),a Personal Digital Assistant (PDA), a cellular telephone, a webappliance, a server, a network router, switch or bridge, or any machinecapable of executing a set of instructions (sequential or otherwise)that specify actions to be taken by that machine. Further, while only asingle computing device is illustrated, the term “computing device”shall also be taken to include any collection of machines (e.g.,computers) that individually or jointly execute a set (or multiple sets)of instructions to perform any one or more of the methodologiesdiscussed herein.

The example computing device 800 includes a processing device 802, amain memory 804 (e.g., read-only memory (ROM), flash memory, dynamicrandom access memory (DRAM) such as synchronous DRAM (SDRAM) or RambusDRAM (RDRAM), etc.), a static memory 806 (e.g., flash memory, staticrandom access memory (SRAM), etc.), and a secondary memory (e.g., a datastorage device 818), which communicate with each other via a bus 830.

Processing device 802 represents one or more general-purpose processorssuch as a microprocessor, central processing unit, or the like. Moreparticularly, the processing device 802 may be a complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,processor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processing device 802may also be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. Processing device 802 is configured to execute theprocessing logic (instructions 822) for performing the operations andsteps discussed herein.

The computing device 800 may further include a network interface device908. The computing device 800 also may include a video display unit 810(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 812 (e.g., a keyboard or touchscreen), acursor control device 814 (e.g., a mouse or touchscreen), and an audiodevice 816 (e.g., a speaker).

The data storage device 818 may include one or more machine-readablestorage media (or more specifically one or more non-transitorycomputer-readable storage media) 828 on which is stored one or more setsof instructions 822 embodying any one or more of the methodologies orfunctions described herein. The instructions 822 may also reside,completely or at least partially, within the main memory 804 and/orwithin the processing device 802 during execution thereof by thecomputer system 800, the main memory 804 and the processing device 802also constituting computer-readable storage media.

The various methods described above may be implemented by a computerprogram. The computer program may include computer code arranged toinstruct a computer to perform the functions of one or more of thevarious methods described above. The computer program and/or the codefor performing such methods may be provided to an apparatus, such as acomputer, on one or more computer readable media or, more generally, acomputer program product. The computer readable media may be transitoryor non-transitory. The one or more computer readable media could be, forexample, an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, or a propagation medium for data transmission, forexample for downloading the code over the Internet. Alternatively, theone or more computer readable media could take the form of one or morephysical computer readable media such as semiconductor or solid statememory, magnetic tape, a removable computer diskette, a random accessmemory (RAM), a read-only memory (ROM), a rigid magnetic disc, and anoptical disk, such as a CD-ROM, CD-R/W or DVD.

In an implementation, the modules, components and other featuresdescribed herein can be implemented as discrete components or integratedin the functionality of hardware components such as ASICS, FPGAs, DSPsor similar devices.

A “hardware component” is a tangible (e.g., non-transitory) physicalcomponent (e.g., a set of one or more processors) capable of performingcertain operations and may be configured or arranged in a certainphysical manner. A hardware component may include dedicated circuitry orlogic that is permanently configured to perform certain operations. Ahardware component may be or include a special-purpose processor, suchas a field programmable gate array (FPGA) or an ASIC. A hardwarecomponent may also include programmable logic or circuitry that istemporarily configured by software to perform certain operations.

Accordingly, the phrase “hardware component” should be understood toencompass a tangible entity that may be physically constructed,permanently configured (e.g., hardwired), or temporarily configured(e.g., programmed) to operate in a certain manner or to perform certainoperations described herein.

In addition, the modules and components can be implemented as firmwareor functional circuitry within hardware devices. Further, the modulesand components can be implemented in any combination of hardware devicesand software components, or only in software (e.g., code stored orotherwise embodied in a machine-readable medium or in a transmissionmedium).

Although physical hardware components have been described, it would beunderstood that the above methods could be carried out in a cloud-basedenvironment and/or on virtualized hardware. As such, this phrase“hardware implementation” used in this section encompasses both physicalhardware or virtualized hardware in a singular function or sharedcomputing environment, which may be located in dedicated equipment spaceor hosted equipment in either private or public cloud.

Unless specifically stated otherwise, as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “receiving”, “determining”,“comparing”, “enabling”, “maintaining,” “identifying”, “providing”,“applying”, “training” or the like, refer to the actions and processesof a computer system, or similar electronic computing device, thatmanipulates and transforms data represented as physical (electronic)quantities within the computer system's registers and memories intoother data similarly represented as physical quantities within thecomputer system memories or registers or other such information storage,transmission or display devices.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other implementations will beapparent to those of skill in the art upon reading and understanding theabove description. Although the present disclosure has been describedwith reference to specific example implementations, it will berecognized that the disclosure is not limited to the implementationsdescribed but can be practiced with modification and alteration withinthe spirit and scope of the appended claims. Accordingly, thespecification and drawings are to be regarded in an illustrative senserather than a restrictive sense. The scope of the disclosure should,therefore, be determined with reference to the appended claims, alongwith the full scope of equivalents to which such claims are entitled.

The invention claimed is:
 1. A computer-implemented method fordynamically training a system to determine an age rating for mediacontent, the method comprising: obtaining age rating data for aplurality of territories, the age rating data comprising a plurality ofage ratings for a respective plurality of content items for a sourceterritory, a target territory, and at least one other territory;determining, based on the age rating data, a similarity vector relatingto the target territory, the similarity vector comprising a predictionscore for each of the source territory and the at least one otherterritory; determining, for the similarity vector, a territoryassociated with a highest prediction score; training, using a trainingdataset, a machine learning model to provide an age rating for a contentitem in the target territory based on an age rating for the content itemin the source territory, wherein the training dataset comprises the agerating data for: the target territory, the territory associated with thehighest prediction score, and, when the territory associated with thehighest prediction score is not the source territory, the sourceterritory; determining, by executing the machine learning model, that aninput content item has a first age rating in the target territory,wherein the first age rating corresponds to a second age rating of theinput content item in the source territory; and in response todetermining that the first age rating exceeds a parental controlthreshold, generating an alert indicative of a parental control when theinput content item is requested for access.
 2. The computer-implementedmethod of claim 1, wherein the input content item is accessed on a mediacontent platform that is accessible in the source territory and thetarget territory, and wherein the alert is generated on a graphical userinterface of the media content platform.
 3. The computer-implementedmethod of claim 1, wherein the parental control threshold and the firstage rating are set in a first rating scale of the target territory, andwherein the second age rating is set in a second rating scale of thesource territory and is incompatible for direct comparison with theparental control threshold.
 4. The computer-implemented method of claim1, further comprising blocking a request to access the input contentitem.
 5. The computer-implemented method of claim 1, wherein the contentitem comprises media content.
 6. The computer-implemented method ofclaim 1, wherein the respective plurality of content items comprisesthousands of content items.
 7. The computer-implemented method of claim1, wherein the at least one other territory comprises a plurality ofterritories.
 8. The computer-implemented method of claim 1, furthercomprising determining the similarity vector by generating a similaritymatrix comprising a plurality of similarity vectors for a plurality ofpairs of territories.
 9. The computer-implemented method of claim 8,further comprising clustering the plurality of similarity vectors intoclusters.
 10. The computer-implemented method of claim 1, furthercomprising determining one or more highest prediction scores by: (a)selecting one or more prediction scores exceeding a threshold, or (b)selecting a predetermined number of prediction scores having a highestvalue, or (c) selecting prediction scores within a set number ofstandard deviations above a mean prediction score.
 11. Acomputer-implemented method for dynamically training a system todetermine an age rating for media content, the method comprising:obtaining age rating data for a plurality of territories, the age ratingdata comprising a plurality of age ratings for a respective plurality ofcontent items for a source territory, a target territory, and at leastone other territory; determining, based on the age rating data, asimilarity vector relating to the target territory, the similarityvector comprising a prediction score for each of the source territoryand the at least one other territory; determining, for the similarityvector, a territory associated with a highest prediction score; inresponse to determining that the territory associated with the highestprediction score is not the source territory, generating a trainingdataset comprising the age rating data for: the target territory, thesource territory, and the territory associated with the highestprediction score; and executing a machine learning model, trained by thetraining dataset, to output an age rating for a content item in thetarget territory based on an age rating for the content item in thesource territory.
 12. The computer-implemented method of claim 11,further comprising: determining, by executing the machine learningmodel, that an input content item has a first age rating in the targetterritory, wherein the first age rating corresponds to a second agerating of the input content item in the source territory; and inresponse to determining that the first age rating exceeds a parentalcontrol threshold, generating an alert indicative of a parental controlwhen the input content item is requested for access.
 13. Thecomputer-implemented method of claim 12, wherein the parental controlthreshold and the first age rating are set in a first rating scale ofthe target territory, and wherein the second age rating is set in asecond rating scale of the source territory and is incompatible fordirect comparison with the parental control threshold.
 14. Thecomputer-implemented method of claim 12, wherein the input content itemis accessed on a media content platform that is accessible in the sourceterritory and the target territory, and wherein the alert is generatedon a graphical user interface of the media content platform.
 15. Thecomputer-implemented method of claim 12, further comprising blocking arequest to access the input content item.
 16. The computer-implementedmethod of claim 11, further comprising, in response to determining thatthe territory associated with the highest prediction score is the sourceterritory, generating the training dataset using the age rating data foronly the target territory and the source territory.
 17. Thecomputer-implemented method of claim 11, wherein the respectiveplurality of content items comprises thousands of content items.
 18. Thecomputer-implemented method of claim 11, wherein the at least one otherterritory comprises a plurality of territories.
 19. Thecomputer-implemented method of claim 11, further comprising: determiningthe similarity vector by generating a similarity matrix comprising aplurality of similarity vectors for a plurality of pairs of territories;and clustering the plurality of similarity vectors into clusters. 20.The computer-implemented method of claim 11, further comprisingdetermining one or more highest prediction scores by: (a) selecting oneor more prediction scores exceeding a threshold, or (b) selecting apredetermined number of prediction scores having a highest value, or (c)selecting prediction scores within a set number of standard deviationsabove a mean prediction score.